[Bugfix][Multimodal] Honor modality-scoped mm_processor_kwargs in every model that reads them - #54527

Follow-up to #53808.

Background

#53808 taught get_merged_mm_kwargs to overlay HF-style nested
images_kwargs / videos_kwargs / audio_kwargs onto the flat namespace, and
threaded a modality argument through Qwen2-VL and Qwen3-VL. Its docstring
states the rule:

When modality is set, HF-style nested images_kwargs / videos_kwargs /
audio_kwargs are overlaid onto the flat namespace for vLLM-side reads
(token budgets, dummy inputs). Processor construction and HF __call__
should omit modality so the nested dicts still reach the HF processor.

Every model outside the Qwen family that predicts sizes or token counts from
those kwargs was left reading the flat namespace only. The HF processor honors
the nested dict in its __call__, so vLLM and the processor disagree about how
many tokens an item expands to.

Scope: every call site audited

Per review this now covers every multi-modal model, not just the VL three.
I enumerated every get_merged_mm_kwargs call site and classified each (40 when
this PR was opened, 44 on current main).

Fixed (vLLM-side reads)

Two are worth calling out:

• mimo_v2_omni is the only one needing a new parameter; it is a copy of
  qwen2_vl.py with the modality plumbing stripped, so it gets the identical
  signature change and two call-site literals.

• cohere_compass landed on main after this PR was opened ([Model] Add Cohere Compass model #54774) with
  the same _get_vision_info shape as Ernie 4.5 VL and Keye, so it is folded in
  here rather than left for a second CI round. It is image-only, so only
  get_num_image_tokens threads modality.

• glm5next landed on main after this PR was opened, with its own copies
  of the GLM-4.1V _get_image_max_pixels / _get_video_max_pixels helpers. I
  first wrote these off as dead code because no call site names them. That was
  wrong: Glm5NextProcessingInfo subclasses Glm4vProcessingInfo and they
  are reached polymorphically, through the get_image_size_with_most_features
  the subclass inherits unchanged. See "Fixing the subclass break" below.

For cohere2_vision / idefics3 / interns1 the merged dict becomes HF's
third positional argument to get_number_of_image_patches(height, width, images_kwargs), which reads it flat (images_kwargs.get("min_patches", ...)). A nested dict arriving there is silently dropped.

Deliberately not changed

• Processor construction and the HF __call__ -- eagle2_5_vl, glm4v,
  h2ovl, internvl (x2), nemotron_vl (x2), nvlm_d, qianfan_ocr,
  skyworkr1v, moss_audio, llava_onevision2. The docstring requires these
  to keep passing the nested dicts through untouched.
• qwen2_5_omni_thinker looks like the block [Bugfix][Multimodal] Honor modality-scoped mm_processor_kwargs #53808 already fixed in
  qwen3_vl with modality="video", and I originally changed it. It must not
  be overlaid. It synthesizes a flat size, and unlike qwen3_vl -- which
  does mm_data.pop("videos") and gives videos their own processor call --
  Omni sends images and videos through one combined HF call. Overlaying a
  nested videos_kwargs makes the guard fire where main skips the block, so
  a flat size built from the video processor's defaults would be applied to
  the images too: strictly worse than main, where HF's own kwarg merging
  scopes the nested dict correctly. Caught in review; reverted, with a comment
  added recording why, since the next person auditing these sites would make
  the same mistake.
• gemma3_mm (x2) feeds the merged dict straight into HF's own
  _merge_kwargs, which routes flat vs nested itself and hands back
  ["images_kwargs"]. Already correct.
• gemma4_mm (x3) and gemma4_unified already hand-roll the nested
  lookup in _get_max_soft_tokens, so there is no defect. Adding the overlay
  would actively break gemma4_mm.py:585: that call uses the helper's second
  return value to distinguish a top-level override from a nested one, and the
  answer is consumed at :732 to decide whether to re-inject the value as a
  top-level kwarg into the HF call. Flattening first would flip that
  decision.

No behavior change without nested kwargs

overlay_modality_mm_kwargs returns its input unchanged when the scoped key is
absent or is not a mapping, so passing modality is a no-op for every existing
configuration. It only starts mattering once a user supplies nested
mm_processor_kwargs, which today these reads silently drop. That is also why
I have not run a model eval: with no nested mm_processor_kwargs the merged
dict is byte-identical before and after.

Reachability

mm_processor_kwargs is a public knob, settable per-server
(--mm-processor-kwargs) and per-request. HF's own processor kwargs are
documented in the nested form, so {"videos_kwargs": {"max_pixels": N}} is the
shape a user copying from transformers docs will write. The HF processor
honors it; vLLM's budget read does not.

Tests

Two model families, both CPU-only, so I ran them end to end on a CPU runner on
my fork.

1. tests/models/multimodal/processing/test_glm4_1v.py::test_videos_kwargs_max_pixels_does_not_leak_into_image_budget
   probes the budgets vLLM computes three times: no kwargs, scoped
   videos_kwargs.max_pixels, flat max_pixels. It pins the two properties
   that make the scoping correct rather than merely different: a scoped video
   override must not move the image budget, and a flat max_pixels must
   still apply to both modalities, preserving the shared-namespace behavior.

2. tests/models/multimodal/processing/test_transformers_image.py::test_scoped_images_kwargs_reach_the_token_count
   covers the Transformers backend on llava-hf/llava-onevision-qwen2-0.5b-ov-hf,
   comparing the per-image token count vLLM predicts under a flat vs a nested
   size override. It asserts up front that the override really moves the
   count, so the test cannot pass by coincidence.

The remaining models take the identical one-line change into the same helper,
and overlay_modality_mm_kwargs itself already has unit coverage in
tests/multimodal/test_processing.py (added by #53808).

Evidence (fork CI run 33432435285, against current main)

Run predates the qwen2_5_omni_thinker revert. That revert only removes a
modality= argument and adds a comment, and neither test touches that file, so
the results below still hold.

GLM-4.1V. On main the scoped override is simply dropped and the video
budget stays at GLM's stock 47,040,000 (the same number the existing
test_get_max_video_frames_matches_glm_resize case in this file uses):

>       assert scoped_video == _SCOPED_MAX_PIXELS
E       assert 47040000 == 469762048

1 failed, 16 deselected, 14 warnings in 13.78s

AFTER, whole file: 17 passed in 27.91s.

Transformers backend. On main the nested override is dropped, so vLLM
predicts the stock token count while the flat form gives the overridden one:

>       assert scoped == flat
E       assert [7329] == [2709]
E         At index 0 diff: 7329 != 2709

1 failed, 22 deselected, 14 warnings in 17.31s

AFTER, whole file: 3 failed, 19 passed, 1 xfailed in 58.64s. The 3 are
pre-existing and environmental, not a regression: all three are
google/gemma-3-4b-it, a gated repo my fork runner has no token for, and they
fail during config download before any vLLM code runs
(GatedRepoError: 401 ... Access to model google/gemma-3-4b-it is restricted).
Flagging them rather than hiding them.

Folded in: per-request mm_processor_kwargs in the Transformers backend

@claude pointed out that the Transformers backend discarded the per-request
kwargs entirely, so the modality scoping above only helped the config-level
ones. Both _get_num_multimodal_tokens reads called
get_merged_mm_kwargs({}, ...) and dropped the hf_processor_mm_kwargs
argument they were handed -- the same argument used to build the HF
processor two lines earlier. A request-level override therefore moved the
processor's output without moving the count vLLM predicts for it. Folded in
here rather than left as a follow-up.

Three pieces:

1. Both reads now merge the real hf_processor_mm_kwargs.
2. OffsetsMultiModalProcessor._get_num_patches_per_image did not receive them
   at all, so it takes them as a parameter and builds its processor with them,
   matching the count path.
3. The merged dict is filtered to what _get_num_multimodal_tokens accepts,
   the way call_hf_processor already filters before splatting the same kwargs
   into __call__. That helper is a sizing method, not __call__, so it does
   not take every processor override.

One trap worth recording. apply() folded its own add_special_tokens=False
into hf_processor_mm_kwargs, and that dict is what reaches the sizing helpers.
Some processors read every kwarg there as an image processor override --
Idefics3 merges them into its images_kwargs defaults -- so forwarding the flag
made every later __call__ fail with
merged_typed_dict.__init__() got an unexpected keyword argument 'add_special_tokens'.
It now lives in a call-only dict. That flag belongs to __call__ alone; the
comment on it already said so.

The same trap exists one key later after upstream's backend refactor:
_get_hf_mm_inputs folds truncation: False into the kwargs it returns, and
_apply_hf_processor_main rebinds its own parameter with that result. So the
request's own kwargs are kept in request_mm_kwargs and those, not the rebound
dict, are what _get_num_image_patches reads.

Evidence (fork CI run 34000087312)

BEFORE is run against 6e70a6a25c, i.e. the rest of this PR without this
commit, so it isolates what this commit fixes. A request-level size override
leaves vLLM predicting the stock count:

>       assert _probe_num_image_tokens(None, request_kwargs) == flat
E       assert [7329] == [2709]

1 failed, 23 deselected, 14 warnings in 17.68s

AFTER, whole file: 3 failed, 20 passed, 1 xfailed.

Whole-file comparison, main vs this branch, so the add_special_tokens
interaction above cannot hide:

The 3 are byte-identical on both sides -- google/gemma-3-4b-it, a gated repo
my runner has no token for, failing at config download before any vLLM code
runs. The 2 extra passes are the two tests this PR adds.

The shared size bound stays unscoped

get_image_size_with_most_features is not scoped, in any of the four models
that have it here. Both review bots flagged the same site: that bound is the
pre-resize upper bound for the image budget, the video frame budget and the
dummy data alike, and smart_resize only rescales an input that already falls
outside [min_pixels, max_pixels]. Scoping it to image therefore let an
images_kwargs-only override shrink the profiled video budget, with nothing
downstream scaling it back up.

It is now read with no modality overlay, with a comment at each site recording
why. The per-item reads (get_num_image_tokens / get_num_video_tokens) stay
scoped and re-resize that bound with the cap for their own modality, so an
override still reaches the count it is meant for. This also keeps the four
models consistent with the reference implementation: #53808's own
qwen2_vl.get_image_size_with_most_features is called unchanged by
_get_max_video_frames / get_max_video_tokens.

Still pre-existing on main and still out of scope here: those two helpers take
their target frame dimensions from the image processor config rather than the
video one. In glm4_1v that mismatch needs no override at all, since
_get_image_max_pixels and _get_video_max_pixels already read two different
processor configs. Fixing it moves stock frame budgets, so unlike everything
else in this PR it is not a no-op without nested kwargs and wants per-model
validation on a GPU. Happy to do it as a follow-up covering the Qwen family too.

Evidence (fork CI run 34183531905, against current main)

BEFORE runs against the previous branch head, since modality scoping does not
exist on main at all. An image-only override drags the shared bound from
2184² down to 700², and the video frame budget inflates from 24 frames to 95:

>   assert (scoped["size_bound"], scoped["video_frames"]) == (
        stock["size_bound"],
        stock["video_frames"],
    )
E   assert ((700, 700), 95) == ((2184, 2184), 24)

1 failed, 17 deselected in 15.27s

AFTER, whole file: 18 passed in 28.40s. Whole-file comparison: main 0
failed / 16 passed, this branch 0 failed / 18 passed.

Fixing the subclass break pre-commit caught

The upstream pre-commit run on 40d876a4 failed, and it was a real regression
this PR introduced rather than a lint nit:

vllm/models/glm5next/nvidia/multimodal.py:647: error: Signature of
"_get_image_max_pixels" incompatible with supertype
"vllm.model_executor.models.glm4_1v.Glm4vProcessingInfo"  [override]

Glm5NextProcessingInfo(Glm4vProcessingInfo) overrides _get_image_max_pixels
and _get_video_max_pixels, but inherits get_image_size_with_most_features
unchanged -- and this PR made that call
self._get_image_max_pixels(modality=None). So the break is not only a typing
one: profiling a GLM-5-Next multimodal model raises
TypeError: Glm5NextProcessingInfo._get_image_max_pixels() got an unexpected keyword argument 'modality'.

The override now takes the same signature and forwards it, the video read is
scoped to "video", and the shared bound is untouched -- identical to
glm4_1v.

mypy only reports a break when the signature changes, so I also swept every
subclass of every processing-info class this PR touches:

The other three inherit the change unchanged.

Evidence

The TypeError was reproduced on fork CI run 34521803748, with BEFORE pinned to
40d876a4 (the previous head of this branch) so that it isolates exactly this
commit:

E       TypeError: Glm5NextProcessingInfo._get_image_max_pixels() got an
        unexpected keyword argument 'modality'

That reproduction used a small regression test, which I have since removed at
@DarkLight1337's request: the [override] error pre-commit's mypy reports is
the same signal, so the test only duplicated a check CI already runs. Whole-file
comparison on tests/models/multimodal/processing/test_glm4_1v.py is therefore
unchanged from the section above -- main 0 failed / 16 passed, this branch 0
failed / 18 passed.

Lint

ruff check, ruff format --diff, typos and the SPDX hook pass on all
changed files. Upstream pre-commit, including the full-tree mypy run that
caught the glm5next override, is green on the current head.

Not a duplicate

Searched open PRs for get_merged_mm_kwargs in:body,
overlay_modality_mm_kwargs in:body, _get_video_max_pixels in:body,
videos_kwargs in:body, images_kwargs in:body,
_get_num_multimodal_tokens in:body, get_number_of_image_patches in:body and
_get_vision_info in:body. Nothing else fixes this.

One overlap to flag: #47876 (open, "Fix video temporal padding token
estimates") also edits glm4_1v.py, keye.py and mimo_v2_omni.py. It is a
different defect -- the temporal_patch_size rounding inside _get_vision_info
-- and touches different lines than the get_merged_mm_kwargs call and
signature, but the two will sit in the same functions. Whichever lands first,
I am happy to rebase.

AI assistance was used to research and draft this change. I have reviewed every
changed line and run the tests above.